1. Field of the Invention
The present invention relates to a method for synthesizing intermediate image using mesh based on multi-view square camera structure and a device using the same and a computer-readable medium having thereon program performing function embodying the same, and more particularly, to a method for synthesizing intermediate image using mesh based on multi-view square camera structure and a device using the same and a computer-readable medium having thereon program performing function embodying the same wherein an accurate disparity vector may be obtained since a shared area is searched for a predetermined time interval using a distance matching and a synthesized image is generated for each area based thereon, an occlusion region is reduced by using three reference images, the synthesized image may be easily generated even for an image having a large disparity, and a converted outline may be accurately expressed, and a high three-dimensional effect may be represented due to the image conversion through the mesh based on a vertex of the outline.
2. Description of the Related Art
A research for a three-dimensional television which is a next generation display of a HDTV is being carried out. A method using a disparity of both eyes has been proposed as an initial type of the 3D television. While the method somewhat provides a three-dimensional effect for an observer, the method does not provide a sufficient vivid motion and reality as in a person seeing an actual object. In order to overcome such disadvantage, a method wherein a more vivid three-dimensional image is displayed using a multi-view image obtained by two or more cameras is under a research.
A moving picture stream used for a multi-view image processing is generally obtained using one camera per view. As an arrangement for a multi-view camera, a parallel arrangement of an in-line type camera, a radical camera arrangement, and the like are available. However, in case of the radical camera arrangement, a distortion of view may occur when a multi-view image obtained from different cameras is displayed on a single screen. The problem may be solved by a parallel camera arrangement method. However, the parallel camera arrangement is disadvantageous in that it is difficult to display a three dimensional effect in a vertical direction. Therefore, a research on a signal processing and a display device employing a square camera arrangement is being carried out recently.
In order to embody a multi-view video system for the 3D television, a transmitter transmits an image obtained from cameras having various arrangements, and a receiver carries out a display based on a transmitted data. In order to transmit the multi-view video through one broadcast channel, a much higher compression ratio is required compared to a compression ratio for a single-view video in video compression. However, it is virtually impossible to transmit an entire multi-view image using a conventional compression algorithm through a conventional broadcast channel. Therefore, an intermediate image synthesis wherein only images the number of which is allowed by the channel are transmitted and the receiver generates a view of image which is not transmitted based on the transmitted image instead of transmitting then entire view of image is under research.
In order to extract a vertex, a feature based method based on a likelihood model, a block based method, a method using a mesh are mainly used as an algorithm related to the intermediate image synthesis. In addition, a method using a depth map is employed. As an example of these algorithms, there are “Multi-view Video System and an Image Generation Method for a 3D Image” disclosed by Korean Patent Application No. 10-2000-0063753 (herein referred to as “Reference 1”), and “Method for Extracting Dense Depth Information using Multiple Camera Input Images and Method for Intermediate Image Synthesis Using the Same” disclosed by Korean Patent Application No. 10-2000-0005297 (herein referred to as “Reference 2”), and these are applied to an application system such as “3D Catalog Shopping System and Method” disclosed by Korean Patent Application No. 10-1995-0042335.
FIG. 1 is a block diagram illustrating a decompression means 34 and an intermediate image synthesizing means 35 disclosed in Reference 1.
In accordance with Reference 1, a method wherein a number of (N−M) images are newly generated between a number of M view-point images obtained in a 3D system using a number of N parallel multi-view images to finally obtain the number of N view-point images is disclosed. For such, an adaptive point extractor 63, and an occlusion region generator 64 for compensating an occlusion region between reference images based on an extracted adaptive point data are used.
Specifically, as shown, nine view-point images are restored in the video decompression means 34 to be input to the intermediate image generating means 35. The intermediate image generating means 35 performs an adaptive point extraction for reference images 0v1, 0v2, 0v4, 0v6, 0v7 and 0v9 from reference images 0v2, 0v4, 0v5, 0v6 and 0v8 in the adaptive point extractor 63, and then extracts and generates the occlusion region based on an adaptive point extraction data in the occlusion region generator 64. Thereafter, a data corresponding to the adaptive point and an interpolated occlusion region are integrated to generate a new intermediate image, thereby obtaining seventeen view-points.
FIG. 2 is a diagram illustrating a conventional MR-MBS (Multi-Resolution Multiple Baseline Stereo) method for synthesizing an intermediate image disclosed by Reference 2.
In accordance with the MR-MBS method of FIG. 2, a depth information of each pixel is densely extracted from images obtained from a finite number of cameras placed in multiple position to generate an intermediate image using this wherein MBS method is converted to a multi-resolution.
Referring to FIG. 2, an input image obtained from multiple cameras with respect to a single object is decomposed into the multi-resolution, and an image having a minimum resolution is subjected to the MBS and the occlusion region process to obtain a depth information of a pixel of the object (step 1) so as to be transmitted to a step 2 wherein a high resolution process is performed. In the step 2, multiple pixels of a high resolution corresponding to a single pixel of a low resolution is subjected to the MR-MBS and the occlusion region process based on the transmitted pixel information to obtain a depth information of the multiple pixels. The process is repeated until a maximum resolution to accurately obtain the depth information for each pixel of the object and the intermediate image.
However, in accordance with the conventional intermediate image synthesizing method, it is difficult to synthesize the intermediate image because the occlusion region is largely generated when a distance between base-lines is large or an angle or a distance between cameras is increased. Moreover, since the adaptive point extractor is applied to an entire object, an unnecessary depth map information and an excessive adaptive point data may be generated.
For example, in accordance with Reference 1, when an intervening image between a left and a right images is synthesized based on a view-point image transmitted from a transmitter in the parallel camera arrangement, a distance between the referred left and the right images are quite increased as a distance of the base-line is slightly increased. As a result, in the synthesis of the intermediate image, the occlusion region is largely increased and the occlusion region cannot be completely compensated using only the adaptive point extractor. Moreover, since the adaptive point extractor is applied to the entire object, an excessive simulation time is required.
In accordance with Reference 2, while problems of a processing time for the MBS and an extension of a boundary line in the depth map are solved by the multi-resolution method employing multiple cameras placed in arbitrary position, the resolution of each pixel and an accurate depth map cannot be obtained when the position and the angle between the cameras are increased. Moreover, in accordance the method, a long simulation time is required due to the unnecessary depth map since the depth map is generated through every camera for the single object and a necessary intermediate image is generated.
In order to overcome the problems, Korean Patent Application No. 10-2004-116703 (not published) filed by the applicant disclose a technology wherein an intermediate image which is not transmitted from a transmitter is synthesized in a receiver using a distance matching, a block matching and an image blending.
FIG. 3 is a flow chart illustrating a conventional method for synthesizing an intermediate image from a portion of a multi-view image based on a multi-view square camera arrangement in accordance with Korean Patent Application No. 10-2004-116703.
As shown in FIG. 3, the method for synthesizing the intermediate image from the portion of the multi-view image based on the multi-view square camera arrangement in accordance with Korean Patent Application No. 10-2004-116703 comprises steps of distance-matching for identifying an overlapping area where an intermediate image to be synthesized and neighboring three reference images overlap (S100 through S110), block-matching wherein a block matching is carried out for a block where a differential image between the reference images in the overlapping area exists (S115 through S130), image-compensation wherein an image of the overlapping area is compensated according to a result of the block matching (S135); and image-synthesizing wherein the intermediate image is synthesized by integrating the compensated image of the overlapping area.
However, Korean Patent Application No. 10-2004-116703 is disadvantageous in that an operation time for obtaining the distance matching is very long and accuracy is low. In addition, each block overlaps when a disparity is obtained from the three images only through the block matching and the image compensation is performed. Moreover, when various algorithms are applied only to an area having a large disparity, it is difficult to maintain an object and a background in the image.
The present invention overcomes the disadvantages of Korean Patent Application No. 10-2004-116703 wherein only the block matching is used for the intermediate image synthesis by perceiving that a more effective 3D image is obtained when outlines of the object and the background are matched based on a mesh.